{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 层和块\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 多层感知机的架构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"../img/5_deep_learning_computation/modelConstruction.png\" width=60%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 事实证明，“比单个层大”但“比整个模型小”的组件更有价值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 由**层组**（groups of layers）的重复模式组成\n",
    "    - 计算机视觉中广泛流行的ResNet-152架构就有数百层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\\begin{definition}\\label{def:block}\n",
    "**块**（module, block）：可以描述单个层、由多个层组成的组件或整个模型本身。\n",
    "\\end{definition}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 块的优势：通过递归可以将一些块组合成更大的组件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "    <img src=\"../img/5_deep_learning_computation/blocks.svg\" width=100% title=\"多个层被组合成块，形成更大的模型\">\n",
    "</center>\n",
    "<p style=\"text-align:center\"><b>多个层被组合成块，形成更大的模型</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\\begin{problem}\\label{prob:blockimplementation}\n",
    "计算机如何实现块？\n",
    "\\end{problem}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 块由**类**（class）表示\n",
    "    - 任何子类都必须定义一个将其输入转换为输出的**前向传播函数**\n",
    "    - 存储任何**必要的参数**（有些块不需要参数）\n",
    "    - 为了计算梯度，块必须具有**反向传播函数**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 在定义自己的块时，由于自动微分提供了一些后端实现，因此，只需要考虑:\n",
    "    - **前向传播函数**\n",
    "    - **必需的参数**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 回顾一下多层感知机的代码：\n",
    "    - 包含一个具有256个单元和ReLU激活函数的全连接隐藏层\n",
    "    - 然后一个具有10个隐藏单元且不带激活函数的全连接输出层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:43.817066Z",
     "iopub.status.busy": "2022-12-07T16:28:43.816409Z",
     "iopub.status.idle": "2022-12-07T16:28:45.216452Z",
     "shell.execute_reply": "2022-12-07T16:28:45.215623Z"
    },
    "origin_pos": 2,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1823,  0.0080, -0.0137,  0.1294, -0.1146, -0.1092,  0.0671, -0.1275,\n",
       "          0.1855, -0.1237],\n",
       "        [-0.1187, -0.1144,  0.0204,  0.1577, -0.1814, -0.1549,  0.1057, -0.0956,\n",
       "          0.2331, -0.1015]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = nn.Sequential(nn.Linear(20, 256), nn.ReLU(), nn.Linear(256, 10))\n",
    "X = torch.rand(2, 20)  # 输入，2个样本和20个特征\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 6,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 通过实例化`nn.Sequential`来构建我们的模型，层的执行顺序作为参数传递\n",
    "- `nn.Sequential`定义了一种特殊的`Module`，维护了一个由`Module`组成的有序列表\n",
    "- 两个全连接层都是`Linear`类的实例，`Linear`类本身就是`Module`的子类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 9,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 自定义块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 11,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 每个块必须具备的基本功能：\n",
    "    1. 将输入数据作为其前向传播函数的参数\n",
    "    1. 通过前向传播函数来生成输出。请注意，输出的形状可能与输入的形状不同。例如，我们上面模型中的第一个全连接的层接收一个20维的输入，但是返回一个维度为256的输出\n",
    "    1. 计算其输出关于输入的梯度，可通过其反向传播函数进行访问。通常这是自动发生的\n",
    "    1. 存储和访问前向传播计算所需的参数\n",
    "    1. 根据需要初始化模型参数\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 12,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 从零开始编写一个块\n",
    "    - 包含一个多层感知机，具有256个隐藏单元的隐藏层和一个10维输出层\n",
    "- 下面的`MLP`类继承了表示块的类\n",
    "    - 我们只需要提供我们自己的**构造函数**（Python中的`__init__`函数）和**前向传播函数**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.220410Z",
     "iopub.status.busy": "2022-12-07T16:28:45.219789Z",
     "iopub.status.idle": "2022-12-07T16:28:45.225103Z",
     "shell.execute_reply": "2022-12-07T16:28:45.224338Z"
    },
    "origin_pos": 14,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    # 用模型参数声明层。这里，我们声明两个全连接的层\n",
    "    def __init__(self):\n",
    "        # 调用MLP的父类Module的构造函数来执行必要的初始化。\n",
    "        # 这样，在类实例化时也可以指定其他函数参数，例如模型参数params（稍后将介绍）\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(20, 256)  # 隐藏层\n",
    "        self.out = nn.Linear(256, 10)  # 输出层\n",
    "\n",
    "    # 定义模型的前向传播，即如何根据输入X返回所需的模型输出\n",
    "    def forward(self, X):\n",
    "        # 注意，这里使用ReLU的函数版本，其在nn.functional模块中定义。\n",
    "        return self.out(F.relu(self.hidden(X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "import torch.nn.functional as F\n",
    "F.relu(input)\n",
    "```\n",
    "- 对输入的张量**按元素**应用relu函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.228240Z",
     "iopub.status.busy": "2022-12-07T16:28:45.227741Z",
     "iopub.status.idle": "2022-12-07T16:28:45.233871Z",
     "shell.execute_reply": "2022-12-07T16:28:45.233130Z"
    },
    "origin_pos": 19,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0225, -0.2324,  0.1964, -0.0633, -0.0183, -0.2165, -0.0312,  0.1983,\n",
       "          0.0844,  0.0095],\n",
       "        [ 0.0370, -0.1302,  0.0604, -0.1831,  0.0357, -0.2166, -0.0843,  0.2079,\n",
       "          0.0165, -0.0483]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = MLP()   # 实例化\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 块的一个主要优点是它的**多功能性**(versatility)：可以子类化块以创建层（如全连接层的类）、整个模型（如上面的`MLP`类）或具有中等复杂度的各种组件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 顺序块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 21,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 构建我们自己的`Sequential`类：`MySequential`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 只需要定义两个关键函数：\n",
    "    1. 一种将块逐个追加到列表中的函数\n",
    "    1. 一种前向传播函数，用于将输入按追加块的顺序传递给块组成的“链条”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.238851Z",
     "iopub.status.busy": "2022-12-07T16:28:45.238121Z",
     "iopub.status.idle": "2022-12-07T16:28:45.244918Z",
     "shell.execute_reply": "2022-12-07T16:28:45.244167Z"
    },
    "origin_pos": 23,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "class MySequential(nn.Module):\n",
    "    def __init__(self, *args):\n",
    "        super().__init__()\n",
    "        for idx, module in enumerate(args):\n",
    "            # 这里，module是Module子类的一个实例。把它保存在'Module'类的成员变量_modules中。\n",
    "            # _module的类型是OrderedDict\n",
    "            self._modules[str(idx)] = module\n",
    "\n",
    "    def forward(self, X):\n",
    "        # OrderedDict保证了按照成员添加的顺序遍历它们\n",
    "        for block in self._modules.values():\n",
    "            X = block(X)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 27,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- `_modules`的主要优点：在模块的参数初始化过程中，系统知道在`_modules`字典中查找需要初始化参数的子块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 29,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 使用我们的`MySequential`类重新实现多层感知机"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.248643Z",
     "iopub.status.busy": "2022-12-07T16:28:45.248107Z",
     "iopub.status.idle": "2022-12-07T16:28:45.254498Z",
     "shell.execute_reply": "2022-12-07T16:28:45.253795Z"
    },
    "origin_pos": 31,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0472, -0.0643, -0.0488, -0.1984,  0.0831,  0.2754,  0.1678, -0.0066,\n",
       "          0.0578, -0.0003],\n",
       "        [ 0.1676, -0.0358,  0.1645, -0.2508,  0.0401,  0.1803,  0.1442, -0.0137,\n",
       "          0.0419,  0.0599]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = MySequential(nn.Linear(20, 256), nn.ReLU(), nn.Linear(256, 10))\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 在前向传播函数中执行代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- `PyTorch`内置的`Sequential`类使模型构造变得简单，允许组合新的架构，而不必定义自己的类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 但，并不是所有的架构都是简单的顺序架构，当需要更强的灵活性时，需要定义自己的块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 有时我们可能希望合并常数参数\n",
    "\n",
    "\\begin{definition}\\label{def:constParam}\n",
    "**常数参数**（constant parameter）：既不是上一层的结果也不是可更新参数的项。\n",
    "\\end{definition}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 33,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "例如，需要一个计算函数\n",
    "\n",
    "$$$f(\\mathbf{x},\\mathbf{w}) = c \\cdot \\mathbf{w}^\\top \\mathbf{x}$$\n",
    "\n",
    "其中$\\mathbf{x}$是输入，$\\mathbf{w}$是参数，$c$是某个在优化过程中没有更新的指定常量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.257787Z",
     "iopub.status.busy": "2022-12-07T16:28:45.257187Z",
     "iopub.status.idle": "2022-12-07T16:28:45.262842Z",
     "shell.execute_reply": "2022-12-07T16:28:45.262094Z"
    },
    "origin_pos": 35,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# 为此，实现一个`FixedHiddenMLP`类，如下所示：\n",
    "\n",
    "class FixedHiddenMLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # 不计算梯度的随机权重参数。因此其在训练期间保持不变\n",
    "        self.rand_weight = torch.rand((20, 20), requires_grad=False)\n",
    "        self.linear = nn.Linear(20, 20)\n",
    "\n",
    "    def forward(self, X):\n",
    "#         X = self.linear(X)\n",
    "        # 使用创建的常量参数以及relu和mm函数\n",
    "        X = F.relu(X@self.rand_weight + 1)\n",
    "        # 复用全连接层。这相当于两个全连接层共享参数\n",
    "        X = self.linear(X)\n",
    "        # 控制流\n",
    "        while X.abs().sum() > 1:\n",
    "            X /= 2  # 在𝐿1范数大于1的条件下， 将输出向量除以2，直到它满足条件为止\n",
    "        return X.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.265695Z",
     "iopub.status.busy": "2022-12-07T16:28:45.265428Z",
     "iopub.status.idle": "2022-12-07T16:28:45.271974Z",
     "shell.execute_reply": "2022-12-07T16:28:45.271209Z"
    },
    "origin_pos": 40,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-0.0247, grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = FixedHiddenMLP()\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 41,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 可以**混合搭配各种组合块的方法**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:28:45.276015Z",
     "iopub.status.busy": "2022-12-07T16:28:45.275750Z",
     "iopub.status.idle": "2022-12-07T16:28:45.285289Z",
     "shell.execute_reply": "2022-12-07T16:28:45.284572Z"
    },
    "origin_pos": 43,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0969, grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面的例子以一些想到的方法嵌套块\n",
    "\n",
    "class NestMLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(),\n",
    "                                 nn.Linear(64, 32), nn.ReLU())\n",
    "        self.linear = nn.Linear(32, 16)\n",
    "\n",
    "    def forward(self, X):\n",
    "        return self.linear(self.net(X))\n",
    "\n",
    "chimera = nn.Sequential(NestMLP(), nn.Linear(16, 20), FixedHiddenMLP()) # 注意数据的形状\n",
    "chimera(X)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
